Papers with multimodal medical tasks
ViLMedic: a framework for research at the intersection of vision and language in medical AI (2022.acl-demo)
Copied to clipboard
Jean-benoit Delbrouck, Khaled Saab, Maya Varma, Sabri Eyuboglu, Pierre Chambon, Jared Dunnmon, Juan Zambrano, Akshay Chaudhari, Curtis Langlotz
| Challenge: | Multimodal medical AI is a growing field of interest, especially for tasks that involve multimodal data. |
| Approach: | They propose a vision-and-language medical library to improve multimodal medical predictions and enable new applications. |
| Outcome: | The vision-and-language medical library aims to improve reproducibility and speed up progress across medical AI . it contains a dozen implementations replicating state-of-the-art results on medical datasets . the library is extensible by researchers but also simple for practitioners . |
Med-MoE: Mixture of Domain-Specific Experts for Lightweight Medical Vision-Language Models (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Recent advances in multimodal large language models have seen remarkable progress for medical decision-making, however, they are designated for specific classification or generative tasks and require model training or finetuning on large-scale datasets with sizeable parameters and tremendous computing. |
| Approach: | They propose a framework that tackles discriminative and generative multimodal medical tasks using multimodal alignment, instruction tuning and routing. |
| Outcome: | The proposed model can achieve superior performance to or on par with state-of-the-art baselines while only requiring 30%-50% of activated model parameters. |